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Abstract. We show that the number of monotone triangles with prescribed bottom 
row (fci, . . . , fc„) G Z", fei < k2 < ... < fc„, is given by a simple product formula 
which remarkably involves (shift) operators. Monotone triangles with bottom row 
(1,2,..., n) are in bijection with n x n alternating sign matrices. 



1. Introduction 

An alternating sign matrix is a square matrix of Os, Is and —Is for which the sum of 
entries in each row and in each column is 1 and the non-zero entries of each row and 
of each column alternate in sign. For instance. 



/ 


1 







1 

-1 1 

1 -1 

1 



1 o\ 

-1 1 



1 

y 



is an alternating sign matrix. In the early 1980s, Robbins and Rumsey (71 introduced 
alternating sign matrices in the course of generalizing a determinant evaluation algo- 
rithm. Out of curiosity they posed the question for the number of alternating sign 
matrices of fixed size and, together with Mills, they came up with the appealing con- 
jecture jH] that the number of n x n alternating sign matrices is 
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This turned out to be one of the hardest problems in enumerative combinatorics within 
the last decades. In 1996 Zeilberger jH] finally succeeded in proving their conjecture. 
Then, some months later, Kuperberg ^ realized that alternating sign matrices are 
equivalent to a model in statistical physics for two-dimensional square ice. Using a 
determinantal expression for the partition function of this model discovered earlier by 
physicists, he was able to provide a shorter proof of the formula. For a nice exposition 
on this topic see p. 
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Alternating sign matrices can be translated into certain triangular arrays of positive 
integers, called monotone triangles. Monotone triangles are probably the right guise of 
alternating sign matrices for a recursive treatment Section 2.3]. In order to obtain 
the monotone triangle corresponding to a given alternating sign matrix, replace every 
entry in the matrix by the sum of elements in the same column above, the entry itself 
included. In our running example we obtain 

/ 1 \ 

10 1 
10 10 1 . 
110 11 

Vl 1 1 1 1/ 

Row by row we record the columns that contain a 1 and obtain the following triangular 
array. 

4 

2 5 

1 3 5 
12 4 5 

1 2 3 4 5 

This is the monotone triangle corresponding to the alternating sign matrix above. 
Observe that it is weakly increasing in northeast direction and in southeast direction. 
Moreover, it is strictly increasing along rows. In general, a monotone triangle with n 
rows is a triangular array (cj,j)i<j<i<n of integers such that aij < ai-ij < ctij+i and 
< fljj+i for all It is not too hard to see that monotone triangles with n rows 
and bottom row {1,2, ... ,n), i.e. a„ j = j, are in bijection with n x n alternating sign 
matrices. Our main theorem provides a formula for the number of monotone triangles 
with prescribed bottom row {ki, k2, . . . , kn) G Z". 

Theorem 1. The number of monotone triangles with n rows and prescribed bottom 
row {ki, k2, . . . , kn) is given by 

( n (id+£.„Ao) n 

\l<p<q<n / l<i<j<n 

where denotes the shift operator, defined by Exp{x) = p{x + 1), and := — id 
denotes the difference operator. 

In order to understand this formula, there are a few things to remark. The product 
of operators is understood as the composition. Moreover note that the shift operators 
commute, and consequently, it does not matter in which order the operators in the 
product Yl (id +-Efcp AfeJ are applied. In order to use this formula to compute the 

l<p<g<n 

number of monotone triangles with bottom row {ki, . . . , kn), one first has to apply the 
operator Yl (id to the polynomial Yl ^{j^^ and then set Xi = ki. 

l<p<q<n l<*<i<'^ 

Thus, it is not so clear how to derive (jl.ll) from this formula. 
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Next we discuss the significance of tlie formula. In the last decades, the enumeration 
of plane partitions, alternating sign matrices and related objects subject to a variety 
of different constraints has attracted a lot of interest. This attraction stems from the 
fact that now and then these enumerations lead to an appealing product formula or 
hypergeometric series, which is, in spite of their simplicity, pretty hard to prove. At 
the moment the search for these simple product formulas seems to be a bit exhausted. 
Therefore, a new challenge is the search for possibilities to give enumeration formulas 
for the vast majority of enumeration problems for which there exists no closed formula 
in a traditional sense. The formula in Theorem ^ contributes to this issue. 

Also note that the second product in the formula in Theorem Q i.e. Yl ^^4-^, 

l<i<j<n 

is the number of semistandard tableaux of shape — n, — n, . . . , ki — 1) and, 
equivalently, the number of columnstrict plane partitions of this shape jHl P- 375, 
in (7.105) g — >■ 1]. In fact, these objects are in bijection with monotone triangles 
with precsribed bottom row {ki, k2, . . . , kn) that are strictly increasing in southeast 
direction j2l Section 5]. Thus, our formula once more gives an indication of the relation 
between plane partitions and alternating sign matrices manifested by a number of 
enumeration formulas which show up in both fields, a phenomenon which is not yet 
well (i.e. bijectively) understood. At this point it is also worth mentioning that we can 
easily rewrite the formula in Theorem Q such that the second product is the number 
of semistandard tableaux of shape (A;„, . . . , fci). 



q=l / l<i<j<n 

n (.d+^CA..A..)) n ^^^1^ 

\l<p<q<n J l<i<j<n 

The method for proving our main theorem can roughly be described as follows. In 
the first step, we introduce a recursion, which relates monotone triangles with n rows 
to monotone triangles with n — 1 rows. This recursion immediately implies that the 
enumeration formula is a polynomial in ki, k2, ■ ■ ■ , k^- In the next step, we compute the 
degree of the polynomial. Finally, we deduce enough properties of the polynomial in 
order to compute it, where the polynomial's degree determines how much information 
is in fact needed. This method is related to the method for proving polynomial enu- 
meration formulas we have introduced in |2] and extended in pi. In the final section 
we mention some further projects around Theorem^ we plan to consider next. 



2. The recursion 

In the following let a(n; fci, . . . , /c„), n > 1, denote the number of monotone triangles 
with (fci, . . . , kn) as bottom row. If we delete the last row of such a monotone triangle 
we obtain a monotone triangle with n — 1 rows and bottom row, say, (/i, ^2, • • • , ^n-i)- 
By the definition of a monotone triangle we have ki < li < k2 < I2 ■ ■ ■ ^ ^n-i ^ 
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< and li ^ Zj+i- Thus 

a{n;ki,...,kn)^ ^ Q;(n - 1; Zi, . . . , Zn_i). (2.1) 



(1, ■■■,(„_!) 
'=l<'l<'!2<-<'=n-l^'n-l^'="''i^'i+l 



We introduce the following abbreviation 

{ki,...,kn) 

E - E 

(il.-.in-l)eZ"-^ (h, .■■,'«-!) 

fcl<il<A:2<---<fc„_i<i„_i<fcn,ijySij_|_i 

for n > 2. This summation operator is well-defined for all {ki,...,kn) G with 
ki < k2 < ■ ■ ■ < kn- Wc extend the definition to arbitrary {ki,...,kn) € by 
induction with respect to n. If n = 2 then 

(kiM) k2 
{h) li=ki 

where here and in the following we use the extended definition of the summation over 
an interval, namely, 

b r/(a) + /(a+l) + --- + /(6) ifa<6 
J2f{i) = \o if6 = a-l . (2.2) 

-f{b + l) - f{b + 2) /(a-1) if6 + l<a-l 



This assures that for any polynomial p{X) over an arbitrary integral domain / con- 

y 

taining Q there exists a unique polynomial q{X) over I such that ^ p{x) — q{y) for 

x=0 

y 

all integers y. We usually write J2 P(^) ^r q{y). (We also use the analog extended 

x=0 

definition for the product symbol.) If n > 2 then 

J2 A(h,...,ln-l) 

{h,...,ln-l) 

(fcl,...,fc„_l) kn (fel,...,fc„_l-l) 

-^{h, ■ ■ ■ Jn-2,ln-l) + -^ih, ■ ■ ■ ,in-2,kn-l)■ 

{ll,...,ln-2) 'n-l=fen-l+l {h,--;ln-2) 

We renew the definition of a{n; ki, . . . , kn) after this extension by setting a{l; ki) — 1 
and 

(fci,...,fc„) 

a{n; ki, . . . , kn) = ^ a{n - l;h, . . . ,ln-i)- 

(h,...,ln-l) 

This extends the original function a{n; fci, . . . , kn) to arbitrary (/ci, . . . , kn) £ Z". The 
recursion implies that a{n; ki, . . . , kn) is a polynomial in ki, ... ,kn- We have used this 
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recursion (and a computer) to compute a{n; ki, . . . , kn) for n = 1, 2, 3, 4 and obtain the 
following 

1, 1 - A;i + k2, \{-'iki + + 2kik2 - klk2 - 2kl + kikl + 3A;3 - Akik^, + klk^^ 

2k2k3-klk3 + kl-kikl+k2kl),^{20k2+nkik2-16klk2+3klk2+4:kikl+3klkl-klkl+ 

Akl-5kikl+klkl-20ks+16kiks-4:klk3-27k2k3+9klk2ks-2kfk2k3-3klklk3+klklk3- 
3k2k3+^kiklk3-klk2k3+16kikl-12klkl+2kfkl-9kik2kl+6klk2kl-klk2kl+9klkl- 

27A;iA;4 + 20^^ A;4 - 3^;^ A;4 + 16k2k4 + 2Akik2ki - 2Aklk2ki + Akfk2k4 - IGkjk^ + 9kiklk4+ 
Skjklki - kfklk^ + 8k^k4 - 6kik^k4 + klk^k4 + Ilk3k4 - 2Akik3k4 + 15klk3k4 - 2kfk3k4- 

9/u2^3^4 ~l~ 2/u2^3^4 '^k^k^ ~\~ ^Jvik^Iv^ Qk-^Iv^k^ ~\~ k-^k^Iv^ ~\~ 3/u2^3^4 /u2^3^4 5/^2^4"!" 

Qkiklk4-klklk4-Ak2klk4+klklk4-2Qkikl+9klkl-klkl+Ak2kl + lbkik2kl-9klk2kl+ 
klk2kl - Ukjkl + Gkiklkl + 2klkl - kik^kl + IGk^kl - lAkik^kj + dk^kl - klk3kl+ 
9/^2 ^3 ^4 ^^2 ^3^4 ^2 ^3^4 ^^3 ^4 3 A^]^ ~|~ 3 A]2 A^^ A^g A^^. H~ A^i A^3 A^4 A^2 A^3 A?4 3 A^]^ A^^ ~|~ 
klkl + 2kik2kl-klk2kl-2klkl + kiklkl + 3k3kl-Akik3kl + klk3kl + 2k2k3kl-klk3kl+ 

From this data it is obviously hard to guess a general formula for (y.{n\ ki, . . . , kn) . 
However, it seems plausible that the degree of Q;(n; /ci, . . . , kn) in /cj is n — 1. In the 
following two sections we prove that this is indeed true. Note that at first glance the 
linear growth of the degree is quite surprising: suppose A{li, . . . , In-i) is a polynomial 
of degree no greater than it! in each of and Zj. Then 

({ki,...,k„) \ 
A{li,...,ln-i) = 
{h,...,ln-l) J 

(ki ki+i \ 
li—i=ki—i li=ki I 

and there exist polynomials A(l\^ . . . , In-i) such that the upper bound 2R+2 is attained. 
Consequently, a{n] ki, . . . , kn) must be of a very specific shape. 

3. Operators related to the recursion 

In this section we define some operators that are fundamental for the study of the 
recursion defined in the previous section. The theory is developed in a bit more gener- 
ality than it is actually needed in order to investigate a{n; ki, . . . , kn)- Recall that the 
shift operator, denoted by E^, is defined as Exp{x) = p{x + 1). Clearly Ex is invertible 
in the algebra of operators of C[x] and we denote its inverse by E~^. Observe that the 
shift operators with respect to different variables commute, i.e. E^Ey — EyE^. The 
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difference operator is defined as A^. = — id. However, tlie difference operator 
Aj, is not invertible since it decreases tlie degree of a polynomial. If we apply the shift 
operator or the delta operator to the i-th variable of a function, we sometimes write Ei 
or Ai, respectively, i.e. AkJih, . . . , A;„) = Aif{ki, A;„). Moreover, A2f{h, h, h), 
for instance, is shorthand for 

i^hfih, k, h)\=k3,i2=k-i,i3=k3 ■ 
The swapping operator Sx,y is applicable to functions in (at least) two variables and 
defined as Sx,yf{x,y) = f{y,x). If we apply Sx,y to the i-th and j-th variable of a 
function we sometimes write Sij. 

In the following we consider rational functions in shift operators. In order to guar- 
antee that the inverse of the denominator always exists, we need the following lemma. 

Lemma 1. Let p{xi, . . . ,Xn) be a polynomial in xi,x^^ ,X2,X2^ ,Xn,x~^ over C, 
and fix an integer i, 1 <i <n. Consider the operator 

id+Afc^p(Efci,Efc2, . . .,EkJ =: Op 

on C[ki, . . . ,kn]. Then Op is invertible and the inverse is 



Op-' = J2i-^yKp(Ek,,Ek„...,E, 



kn) 1 



1=0 



where A°.p(i?fc^, E^^, ■ ■ ■ , E^^)^ = id. Moreover 

degfc^ G{ku ...,kn) = deg^^. Op . . . , = deg^^^ Op"^ G{ki, . . . , /c„). 

Proof. Let G{ki, . . . ^kn) G C[/ci, . . . , /c„]. First observe that 

degfc, G{ki, . . . , fc„) = degfc^ Op G{ki, ...,kn), (3.1) 

since A^^ decreases the degree in ki and p{Ekj^, Ek^, ■ ■ ■ , Ek„) does not increase the 
degree. It is easy to see that 

oo 

E{k,, . . . , fc„) = J2i-^y^iAEk,,Ek,, EkjG{k,, ...,kn) 

1=0 

is a polynomial with the property that OpF = G. (Observe that the sum is finite 
since A[_G{ki, . . . , /c„) = if / > deg;,. G.) Assume there is another polynomial F' G 
C[ki,...] kn] with the property that Op F' = G. Then Op if = with H = F - F' . 
Thus, by (jSUl), deg^. H = deg^. = — oo, a contradiction. We obtain degjr.. Op^^ G = 
degfc, G if we apply (jni) to Op""^ G. " □ 

Next we define two operators applicable to polynomials G{ki, . . . , kn) G C[ki, . . . , kn]- 
We set 

\4„fc, = id+E,-/Afe^Afc^, = E,-i(id+Efc^AfcJ 
and ^ 
Tk„ki+i = {'^(^+Ek,+iEZ^Sk„k,+i^ " 



By Lemma m the inverse (Vk^^ki+i + Vfc.^^^^fcJ^^ is well-defined. The following lemma 
explains the significance of Tk-^ki+i for the recursion ()2.1|) . 
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Lemma 2. Let A{li,l2) be a polynomial in li and I2 which is of degree at most R 
in each of li and l2- Moreover assume that Ti-^^i^A{li,l2) is of degree at most R as a 
polynomial in li and I2, i.e. a linear combination of monomials l^l2 with m + n < R. 
Then 

A{h, l2)=J2Yl ^(^1' ^2) - Mh, k2) 

{liM) h=kil2=k2 

(ki,k2,k3) 

is of degree at most R + 2 in ^2. Moreover, if Ti^^i2A{li,l2) = then ^{hyh) is 

{h,h) 

of degree at most R + 1 in k2- 

Proof. We decompose A{li,l2) = Ti^,i2^{h,h) + {id—Ti^^i2)A{li,l2)- If we define 
A*{li,l2) = (id — T/^^/2)((/i)p(/2)g/(p!5'!)), it suffices to show that the degree of 

k2 ks 

5^ A*(/i,/2)-A*(A;2,A;2) (3.2) 

h=ki l2=k2 

in k2 is no greater than max(p, g) + 1, where {x)p = 11 + 0- Observe that 

i=0 

id-^Zi,«2 = TP 7^7 — (id--Ez2-E^7^5'ii,«2), 

^h,l2 + ^l2,h 

where we use the fact that Si^^i^R{Ei^, Ei^) = R{Ei^, Ei^)Si^^i^ if R{x,y) is a rational 
function in x, a;~^, y, y"^. Lemma Q imphes that 



-(M.4gi:H)'(;) 



X 



i=0 j=0 

{h + i - j)p-i{k + j)q~i _ ik +j + l)p-i(/i + i-3 - l)g-» 
{p — i)\{q — i)\ {p — i)\{q — i)\ 

ih+i- j + l)p-i-l{l2 + j)q-i-l {h +3 + l)p-i-l(/l + i - j)q-i-l 



(p - z - l)!(g - z - 1)! {p-i- l)!(g - i - 1)1 

Using the summation formula 

^ 1 

YU + W)n = — ((& + W)n+1 - (a - 1 + W)n+l) (3.3) 
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we observe that ()3.2|) is equal to 



i=0 j=0 \ / \-J 



X - 



h + P ~ j\ fk2 + q + j - i - l\ ^ A2 + g - j - 1\ /'/C2 + P + j - i 
p-i+lj\ q-i+1 J \ q-i+1 J\ p-i+1 

h + P - j\ fh + q + j - i - _^ fk2 + q - j - l\ fk2+ p + j - i - I 
p—i J\ q—i J \ q—i J\ p—i 

k2+ P - j - '^\ [k2 + q + j ~ i - l\ ^ fk2 + q- j-2\fk2+p + j- i 



p — I J \ q — I J \ q — I J \ p — I 

k2+p-J-l\fk2 + q + J-i-2\^fk2 + q-]-2\fk2+p + j-i-l 
p — i — 1 J \ q — i — 1 J \ q — i — 1 J \ p — i — 1 

+ R{ki,k2,k3), 

where R{ki, k2, k^) is a polynomial in fci, ^2, ^3 of degree no greater than max(j9, q) + 1 
in ^2- If we replace j hy i — j in every other product of two binomial coefficients we 
see that this expression simplifies to R{ki, ^2, k^) and the lemma is proved. □ 

In order to use Lemma El to compute the degree of J2 ^{h,h) in ^2, one has 

to compute the degree of Ti-^^i^A{li,l2) in ^1 and l2- However, the operator T^^ ^j is 
complicated, and thus it is convenient to consider a simplified version of T;^ for this 
purpose, which is obtained by multiplication with an operator that preserves the degree. 

{id+Sk,,k,+jEk,Vk,M+i = (id+5fc„fc,+J(id+Efc,+iAfcJ 
Observe that deg^^^^^^^ Tfc^,fc^^,G(fci, ...,kr,) = deg^^^^^^^ • • • > ^n)' since 

and Ajfc-Afc.^i decreases the degree of a polynomial in ki and fcj+i. In particular, 
Tk„k,+iG{ki, ...,kn) = if and only if T^^. ^G(/ci, . . . , = 0. 



4. The fundamental lemma 

Suppose A{li, . . . ,ln) is a function on Z". In this section we prove a lemma that 
expresses 

((fci,...,fc„+i) \ 
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in terms of T[^__^i^A{li, . . . ,ln) and T/. . . . , /„). In particular, this shows that if 

• • • , ^n) = for alH = 1, . . . , n — 1 then 

((fcl,...,A;„+l) \ 
{h,-M J 

for all i = 1, . . . , n. 

Lemma 3. Let f{ki, k2, ks) be a function from 1? to C and define 

g{ki,k2,k^,ki) -.^ ^ fih.h.h)- 

Then 

T2,3 9{h,k2,ks,k4) = 

-. / ks ks ki fe2+l ks—l fca — 1 \ 

\ll=k2+l l2=k2 + l l3=k2 h=kl l2=k2 l3=k2 / 

/fcs-l fca-l fca-l fc3-l 

+ 9 E E M^d+Ei)Tl2f{h,l2,k2)- Yl E M^d+E,)Ti^,f{k2 + l,l2,k 



+ I {TI2 fih, k2, k2 + l)- TI2 fih, k2, h + l)+ T^,3 fik2, k2, ^2) - T^,3 fiks, k2, k2)) 

- r;^2 fih, ks, k2+l)- fik2, k2, ks). 

Moreover, for a function h{li, I2) on 1? , 

({klMM) \ , fe2-l fc2-l 

E hih,l2) ih,k2,h) = -2 E E T[,2hih,l2). 
{h,l2) ) h=ki h=ki 

Proof. We only sketch the proof of the first formula since the proof of the second 
formula is easy. Observe that in this formula T"^' 2/(^2, ^2, ^2 + 1), for instance, is 
shorthand for 

TUfih,h,h + l)l^^,^,^^,^. 

By definition 

k2 k[i k4 k4 k'z 

giki,k2,ks,ki) = E E Y fik,k,h) - Y fik2,k2,h) - ^ fik,ks,ks). 

h=k\ l2=k2 h=k3 h=kz h=ki 

It is easy to see that 

fc2 — 1 fc4 fcs ki 

^29ikl, k2, ks, A;4) = - E E -^(^1' ^3)+ E E -^(^2 + 1, k, k)-fik2 + l, ks, ks). 

Il=kll3=k3 I2=k2+2l3=k3 
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This implies that 

k2 fca fc4 

{id+A2E^)g{h,k2,ks,h)= J2 E T.fihj2j3) 

h=ki h=k2 h=k3 

- k2, h) - /(^l' ^3, k^) - f(k2 + l,ks + 1, ks + 1) 

(3=^3 h=l<'i 

k2 — l k4 fca+l k4 

-EE /(^l'^2,/3)+ E E /(^2 + l,/2,/3)- 
h=kll3=k3+l l2=k2+2 l3=k3+l 

Next we want to apply the operator (id +6*2,3) • Observe that 

(k2 fca k4 \ 
E E E/(^l'^2,/3))(fcl,fc2,fc3,M 
ll=kl l2=k2 l3=k3 / 

k2 fca k^ k'i fca fc4 

- E E E /(^i'^2,/3) - E E E fih,i2,h) 

h=kl l2=k2 l3=k3 li=ki l2=k2 l3=k2 

k3 k3 k4 k2 ks — l fe — 1 k2 ^4 

= - E EE /(^i' ^3) - E E E /(^i' ^3) + E E /(^i' ^3) 

il=fc2+l i2=fc2 + l ^3=^2 h=kl l2=k2 l3=k2 ll=kl (3=^2 

ks k4 fca fca-l 

+ E E/(^l'^3,/3)+ E E/(^l'^3,^3) 

'l=fcl '3=^:3 il=fc2 + l ^3 = ^2 

-. / k3 fea fc4 fc2 fca-l fca-l 

= "2 E E E(id+^l,2)/(il,/2,/3)+E E E0d+'^2>3)M,i2,i3) 
\(i=A;2+l i2=A;2+l l3=k2 li=ki l2=k2 l3=k2 

k2 ki fca fc4 fca fea— 1 

+ E E /(^i' ^3) + E E /(^i' ^3, ^3) + E E /(^i' ^3, ^3). 

'i=fei ^3=^:2 h=ki l3=k3 /i=A;2+l ^3=^2 

Therefore, we have 
(id +62,3) (id +A2Es)g{ki, k2, k^, k^) 

/ fca fca fc4 fc2+l fca-1 fca-1 ^ 

= -2 E E E^l'2/(^l'^2,/3)+E E E^2,3M,^2,/3) 

\'l=fc2+l l2=k2 + l l3=k2 ll=k\ l2=k2 l3=k2 

fca fca+1 fca-l 

+ E f(^^ + 1, ^2 + 1, ^3) - E ^(^2 + l,ks+ 1, /3) - E ^(^2 + 1, k: ks) 
l3=k2 h=k2 l2=k2 

k3 + l fc3 — 1 ks 

+ E /(^3 + 1,^2,^:2)+ E fih,h,k2)- E fih,k2,k2) 

l2=k2+2 h=k2+2 h=k2+2 

- f{k2: k2, ks) - f{ks + I,k2 + 1, k2 + 1). 
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Finally check that the right-hand-side of this equation is equal to the right-hand-side 
in the statement of the lemma. □ 

This proves the statement preceding the lemma for n = 2,3. It can easily be extended 
to general n by deriving a merging rule for the recursion ()2.H) . For this purpose we need 
another operator. Let f{x,z) be a function on Z^. Then the operator J^^ transforms 
f{x, z) into a function on Z by 

IIJ{X, Z) := f{y - 1, + /(y, y + l)-f[y-l,y + l) = V,,J{x, ^) 1,=,,,=, . 

With this definition we have 

(fcl,...,fcn) {ki,...,ki^i,k'.) {k",ki+i,...,kn) 

J2 A(/i,...,/„.o = 4%r E E Mh,...Jn). (4.1) 

{ll,...,l„-l) {li,...,l„-l) 

Fix a function A{li, ...,/„) on Z" and an i with 2 < -j < n — 1. Let 

{x,ki,ki+i,y) 
(Zi_i,Zi,ii+i) 

and 

(fci,...,A:i_2,to) (2,A:i+3,...,fc„+i) 
^«),a;,y,^(^l5 • • • 5 ^*-2) ^i) • • • ; ^n+l) = ^x,y (^1 5 • • • 5 • 

Then, by dHH), 

(fci,...,fe„+i) 

Define 

-2 E E E ^-1.^(^1'- ••'^")+ E E E 

\ ^i — 1 — ^iH~l — ^z + 1 — — ^ — — 

^ I fci+i-i fci+i-i 

j+2, . . . , 

\ — \ — li — k-i 
ki+i-l 

- ^ ^ Aj(id+^i+i)7yj_^iv4(/i, . . . ,/i_2, fcj + . . . ,/„) 

li — — ki 

2 ■ ■ 1 U-2-, ki, ki, ki + 1, /j+2, • • •) ~ • • 5 ^i-2; ^j; ^ij + 1; ^i+2; • • •) 

+ T■^^_^_^A{. . . , /j_2, fcj, h, ki, /j+2, • • •) — T/j^iA(. . . , /j_2, fcj+i, h, ki, Zj+2, . . .) j 

— jv4(. . . , /j_2, fcj, /Cj+i, ki + 1, /j+2, • • •) ^ ■ • 5 ^i-2; ki, ki, fci+l, /i4.2, • • •) 



!)•••) 'nj 
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and 

{ki,...,ki^2,w) {z,ki+3,...,kn+i) 
^w,x,y,zi.kiy ■ ■ ■ y ki-2j ki, ki^l, ki^3, . . . , kn+l) = A* ,^ (/l ,...,/„) . 

{ll,...,li-2) ik+2,---,l-n) 

Then, by the first formula in Lemma El we have 

((fci,...,fe„+i) \ 
n+l) 

~ ^w,x^ -^y'z^ ^w,x,y,zikl^ • • • , ^4-2, ^i, ^j+l, • • • ; ^n+l)- (4-2) 

If we use the second formula in Lemma El we obtain a similar formula for the case 
i = 1. By symmetry an analog formula follows for i = n. These formulas imply the 
following corollary. 

Corollary 1. Suppose A{li, . . . ,ln) is a function on Z" with Tl_ i,_^^A{li, ...,/„) = 
for all i, 1 < i < n. Then 

({ki,...,k„+i) \ 
J2 A{h,...,Q \ (fci,...,A;„+i) = 
{ll,...,ln) J 

for all i, I < i < n. 

We come back to /ci, . . . , kn). By induction with respect to n we conclude that 
i(^iP''^ki, . . . ,kn) = for all i, 1 < 2 < n, if n > 2. (Note that a{2;ki,k2) = 
k2 — ki + 1.) Thus Tfc. fc.^ja(n; ki, . . . , = for all i. Therefore, by Lemma |21 and 
by induction with respect to n, the polynomial a{n; ki, . . . , is of degree no greater 
than n — 1 in every k^. 

5. Proof of the theorem 

In the previous two sections we have seen that the fact that Tl^, i^_^_^a{n; ki, . . . , kn) = 
for all i is fundamental for the computation of the polynomial's degree. In this section, 
however, we demonstrate that this property already determines a{n; ki, . . . , kn) up to 
a multiplicative constant. Observe that T^^ ^^(^1, • • • , kn) = is equivalent with the 
fact that (id Afc-)A(fci, . . . , is antisymmetric in ki and fcj+i. In the following 
lemma we characterize functions A{ki, . . . , with the property that 

(id+Efc^+,AfcJA(A;i,...,fc„) 

is antisymmetric in ki and /cj+i for all i. 

Lemma 4. Let A{ki, . . . , kn) be a polynomial in {ki, . . . , Then 

{id+Ek^^^AkJA{ki,...,kn) 

is antisymmetric in ki and ki^i for all i, 1 < i < n — 1, if and only if 



I Yl {id + Ek^ Ak U Aik^,...,r.n) 
\l<p<q<n / 
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is antisymmetric in fci, . . . , 

Proof. First assume that (id H-i^fc^^j Afc.)y4(fci, . . . , kn) is antisymmetric in fcj and fcj+i 
for all i. We have to show that 

(id+^fc.A+i)( n (id+i^^fc,Afcjj A(/ci,...,M=0 

\l<p<q<n / 

for all i. For this purpose observe that 
(id+^fc„fc,^J n (id+^fc,AfcJ {id+Ek,^,AkJA{ki,...,kn) 

\l<p<q<n,{p,q)yt{i,i+l) J 

Yl (id+Efc,AfcJ {id+Sk,,k,^,)iid+Ek,^,AkJA{ki, . . . ,kn) = 0, 

^l<p<q<n,{p,q)9^ii,i+l) ) 

because 

/ \ 



J] (id+E,,^Ai 

l<p<g<n,(p,g)7^{j,i+l) 



l<p<q<n, 
\p,q^{i.z + l} / 



X 



n (id+E,,A,,^J j ( n (id+i?fe,A,J j ( n (id+i?..+,AfcJ 

-l<q<n / \l<p<i / \1<|3<« / 



\i-\-\<q<.n J \\<p<i J \l<p<i 

is symmetric in ki and fcj+i. Conversely, assume that 



Yl (id+Ek^Ak^)] A{ki,...,kr^ 

vl<p<g<n / 



is antisymmetric in /ci, . . . , /c„. Consequently, 

J] (id+Efc,AfcJ (id+5fc,,fc,^J(id+Efc,^,AfcJA(/ci,...,A;„) = 0, 

^l<p<g<n,(p,(j)7^{i,i+l) / 

for all i, 1 < i < n — 1. By Lemma ^ the operator ]^ [id+E^ A^ ) is invertible, 

1 <p<q<n, 
(p,q)#(i,i+l) 

and therefore (id+S'fc^,fe^_^J(id+Efc^^^AfcJA(A;i, . . . , = 0. □ 
Using this lemma we see that 



Y[ (id+Efc,AfcJ j a(A;i,...,A;„) 

vl<p<g<n / 



(5.1) 



is an antisymmetric polynomial in fci, . . . , A;„. A product of shift operators does not 
increase a polynomial's degree, and thus the degree of ()5.1|) in every ki is no greater 
than n — 1. Every antisymmetric function in fci, . . . , /c„ is a multiple of Yl (^i ~ ^i)) 

l<i<j<n 

and since this product is of degree n — 1 in every ki, the expression in ()5.H) is equal to 
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^ n (^i ~ where C is a rational constant. By Lemma ^ Y\ {id +E^^/S.^^) 

l<i<j<n 1<P<9<1 

is invertible, and therefore 



a{n; fci, . . . , K, 



n idT^Ar)^ n ft-*^. 

;p<q<n i^q i^v / l<i<j<n 



vl<p<g<n '"9 '"P / l<i<j; 

We compute the constant C. We expand a{n; ki, . . . , kn) with respect to the basis 

n 

Y[i^i)mi and consider the (non-zero) coefficient of the basis element with maximal 

i=l 

(m„,m„_i, . . . ,mi) in lexicographic order. We show by induction with respect to n 

n 

that (m„, m„_i, . . . , mi) = — 1, — 2, . . . , 1, 0) and that the coefficient is Yl {i-iy. ■ 

i=l 

Assume that the assertion is true for — 1. A careful analysis of the definition of 

(fei,...,fc„) 

^ shows that the "maximal" basis element of a{n; ki, . . . , kn) with respect to 

(h,---,ln-l) 

the lexicographic order is the "maximal" basis element of 

E E ••• E n 



1 «„_l=fc„_l+l i=l ^ ' 



h=ki h=k2 + 

n 

The assertion follows and thus C = H JJ^TJi ~ Yl 73^- We obtain the following 

i=l l<*<i<" 

Theorem. 

Theorem 2. The number of monotone triangles with n rows and prescribed bottom 
row {ki, k2, ■ ■ ■ , kn) is equal to 

\ TT ~ 

id+Ek Ak j -i ' 

a<P<9<n " ^ J l<i<j<n •' 

By the formula for the geometric series, the inverse of the operator id+-Efc^Afcp 
appearing in this formula is equal to 



/=0 

This follows from the proof of Lemma However, it is also possible to give a similar 
formula for a{n\ ki, . . . , kn) which does not involve inverses of operators. In order to 
derive it, we need the following lemma. 

Lemma 5. Let P{Xi, . . . , X„) be a polynomial in {Xi, . . . , Xn) over C which is sym- 
metric in {Xi, . . . , Xn). Then 

p{Ek.,...,E,j n ^=p{i,.--A)- n 



l<i<j<n ^<i<j<n 
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Proof. Let (mi, . . . , m„) G be with mj > for all i and mj 7^ for at least one i. 
It suffices to show that 

' ' ' fcn J. J. „■ ^ 

7re5„ l<«<i<" 

By the Vandermonde determinant evaluation, we have 

TT k^zh = det (( 
ii<l<n ^ - ' V VJ - 1 

Therefore, it suffices to show that 

El ki \ I \ I k„ 



7r,CTG<S, 



If, for fixed vr, cr G 5„, there exists an i with cr(i) — m,r(j) — 1 < then the correspond- 
ing summand vanishes. We define a sign reversing involution on the set of non-zero 
summands. Fix vr, cr G 5„ such that the summand corresponding to tt and a does not 
vanish. Consequently, {^(l) — m^(i) — 1, cr(2) — m^(2) — 1, ... , a{n) — mT,(^n) — 1} ^ 
{0, 1, . . . , — 1} and since (mi, . . . ,m„) 7^ (0, . . . , 0) there are ^, j, 1 < i < j < ri, 
with a{i) — niT^^i) — 1 = a{j) — m^(j) — 1. Among all pairs with this property, 
let be the pair which is minimal with respect to the lexicographic order. Then 

the summand corresponding to vr o and a o is the negativ of the summand 

corresponding to vr and a. □ 

Observe that Yl (1 + ^gi^p ~ 1)) is symmetric in (Xi,...,X„). Thus, by 

l<p,g<n 

Lemma 

n (id+£.,A.,) n ^= n 

i<Piij<" i<*<i<" i<j<i<" 

Therefore, by Theorem |21 



a{n; fci, . . . , fc„; 



n id ) n . ^ 

vl<p<g<n "f/ l<i<i<n 

n ijTibr) ( n (id+^^-Ao) n 



n (id+i^.A,j] n 

vl<p<g<n / l<^<j<n *^ 



and this completes the proof of Theorem ^ 



6. Some further projects 

In this section we list some further projects around the formula given in Theorem 
we plan to pursue. 
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(1) A natural question to ask is whether it is possible to derive the formula for the 
number of n x n alternating sign matrices from Theorem ^ i.e. to show 
that 



^l<p<^<n 




=n 

iki,k2,-,kn)=(l,2,...,n) 



(3j-2)! 
n+j-l)\ 



More generally, one could try to reprove the refined alternating sign matrix 
theorem JU], which states that the number of n x n alternating sign matrices 
in which the unique 1 in the top row is in the k-th column is given by 

+ (3j-2)! 

(n-1)! J-jL ^n + j-iy: ^ ' 

An analysis of the correspondence between alternating sign matrices and mono- 
tone triangles shows that a(n — 1; 1, 2, . . . , — 1, /c + 1, . . . , n) is the number 
of n X n alternating sign matrices in which the unique 1 in the bottom row 
is in the /c-th column and this is by symmetry equal to (jG.lj) . This could be 
a consequence of an even more general theorem: computer experiments sug- 
gest that there are other (fci, ^2, . . . , fcn) G "near" (1,2, ...,n) for which 
Q;(n; /ci, . . . , fc„) has only small prime factors. Small prime factors are an indi- 
cation for a simple product formula. A similar phenomenon can be observed 
for some (fci, ^2, . . . , fcn) £ Z" "near" (1, 3, . . . , 2n — 1). It is not too hard to see 
that a{n\ 1, 3, . . . , 2n — 1) is the number of (2n + 1) x (2n + 1) alternating sign 
matrices, which are symmetric with respect to the reflection along the vertical 
axis. Kuperberg [S] showed that the number of these objects is 

n\ (6j-2)! 



(2ra)!2" (2n + 2j 



-V. 



(2) Let j3{n; ki, . . . , kn) denote the number of monotone triangles with prescribed 
bottom row (fci, . . . , fc„) that are strictly increasing in southeast direction. With 
this notation. Theorem ^ states that 



a(n; ki, . . . ,kr, 



II {id+Ek^Ak^j (3{n;ki,...,kn). (6.2) 

\l<p<q<n / 



It would be interesting to find a bijective proof of this formula in the following 
sense: if we expand the product of operators on the left hand side we obtain a 
sum of expressions of the form 

EtlEtl ■ ■ ■ Et:K\Kl ■ ■ ■ KlPin; k„ . . . , fc„) 

with ai,bi G {0,1,2,...}. We can interpret these expressions as sums and 
differences of cardinalities of certain subsets of monotone triangles with n rows. 
For instance, 

Ak3{n; ki,...,kn} 
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is the number of monotone triangles that are strictly increasing in southeast 
direction and with bottom row {ki, . . . , kg + 1, . . . , kn) such that the (g — 1)- 
st part of the (n — l)-st row is equal to kg minus the number of monotone 
triangles that are strictly increasing in southeast direction and with bottom 
row (fci, . . . , kn) such that the g-th part of the (n — l)-st row is equal to kg. 
In order to prove ()(j.2|l . one has to show that these cardinalities add up to the 
number of monotone triangles. Equivalently, one could follow a similar strategy 
for the identity 



which is equivalent to Theorem |2l 
(3) This is more a remark than another project: to prove Theorem ^ I have more 
or less carried out an analysis of the recursion ()2.H1 . I originally started this 
analysis when considering a somehow reversed situation: let an (r, n) monotone 
trapezoid be a monotone triangle with the top n—r rows cut off and bottom row 
(1, 2, . . . , n). Let -y{r, n]ki,..., kn-r+i) denote the number of (r, n) monotone 
trapzoids with prescribed top row {ki, . . . , kn-r+i)- In particular, 'y{n,n; k) is 
the number of monotone triangles with n rows, bottom row {1,2, ... ,n) and 
k as entry in the top row. In the bijection between alternating sign matrices 
and monotone triangles, the entry in the top row of the monotone triangle 
corresponds to the column of the unique 1 in the first row of the alternating 
sign matrix. Thus, 'j{n,n;k) must be equal to (jfi.lj) . On the other hand, we 
can also use ()2.H) to compute 7(r, n;ki,..., kn-r+i)'- 7(1, n;ki,..., kn) = 1 and 




(l,fcl,...,fc„_r + l,n) 

'y{r,n;ki, . . . ,kn-r+i) = ^ 7(r - 1, n; Zi, . . . , /„_^+2)- 

{ll,...,ln-r + 2) 
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With this extended definition, 'y{n,n; k) is a polynomial in k. In the following 
we list it for n = 1, 2, ... , 6. 

7(l,l;fc) = 1 

7(2, 2; fc) = -1 + 3A;-P 

7(3, 3; fc) = -^(48 - 92A;+103A;2-40fc3 + 5fc^) 

7(4, 4; A;) = ^^(-2160 + 5910 A; - 5407 A;^ + 2940 fc^ 

-919A;^ + 150P-10A;^) 

7(5, 5; fc) = ^^(584640 - 1644072 A; + 1970008 A;^ 

-1211172 k^ + 456863 k^ - 111708 k^ 
+17462 A;^ - 1608 A;^ + 67 A;®) 

7(6, 6; A;) = -^(-73316880 + 225502200 A; 
7560 

-284097336 A;^ + 204504097 k^ 
-91897169 k^ + 27466950 k^ 
-5651016 A;'' + 805518 A;^ 
-77646 A;* + 4655 k^ - 133 A;^°) 

Unfortunately, these polynomials are not equal to 1)6.11) . (For instance, they do 
not factor over Z.) They only coincide on the combinatorial range {1, 2, . . . , n} 
of k. However, it might still be possible to compute 'y{n,n; k) for general n. 

Strikingly the degree of 'y{n,n;k) in A; is 2n — 2 as the degree of ()6.H) . 
This linear growth is again unexpected because the application of ()2.1|) can 
more than double a polynomial's degree, see Section El However, one can use 
Lemma 121 and an extension of Lemma El to show that, more generally, the degree 
of 7(r, n]ki,..., kn-r+i) is 2r — 2 in every ki. 
(4) Finally we have started to investigate a g- version of the formula in Theorem ^ 
i.e. a weighted enumeration of monotone triangles with prescribed bottom row 
{ki, . . . , kn) which reduces to our formula as q tends to 1. 
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